Last updated on December 10, 2025
First of all, I’d like to name the multimodal token of physics pixel in previous posts as OmniToken, because Grok tells me that it is in fact different from all tokens before and especially from those called multimodal tokens. As Grok’s opinions, this OmniToken is unique for only one token type in the entire model, separate orthorgonal tables for different data fields (text, physics, etc), etc, so it’s omni for it can include everything at least for now. And below, lets talk about maybe my last piece for this OmniToken.
Maybe I should hold this last piece for some fat offer from some book company? hahaha. Alright, this is a test, all you guys working on this OmniToken without putting interface ID in the parameters of physics pixel fail this test, See you next semester in this very course again!
Haha, in fact I forgot to add interface ID in the OmniToken in previous posts. But later I realized that interface is a very fundamental component for physics pixel concept, not only for a physics pixel is a point on an interface between 2 objects, but also for interface ID is the basis for the AI model (like transformer) to understand the features of an 2D interface or surface (like fluids layers or organ surface or text on sign/book), which is necessary for all applications.
The necessity for interface ID is that interface ID can identify or label the scope/group of physics pixels different from object ID, which is necessary for case like a sign painted on road or front/back pages of a sheet. And interface ID shall be in different format from object ID by different length of bits, prefix/supfix or other formats.
An interface could be a shared 2D boundary formed by 2 or more surfaces of 2 or more adjacent objects, or formed by a surface of one object with one or more surface of one or more other object. So an interface or an interface ID includes or represents at least 2 adjacent surfaces of at least 2 objects. So we shall use both interface ID and object ID to identify a surface, or an surface ID includes an interface ID and an object ID of the object which the surface is on.
When we describe a feature of boundary of more than one surfaces as one interface like refractive index, the object ID can be set as “0”.
And also interface/surface can have hierarchy structure too, like one surface of one object is a parent interface/surface and its part contacting with another object can be its child interface/surface.
So we need to add interface ID or surface ID to physics pixel for OmniToken in all applicatons, to let the model can learn, understand and generate the features of an 2D interface or surface in a 3D physics pixel frame. For most applications, the features of an interface could be very various and without fixed format, so the features of an interface can be added to text table of the tokens, and for some specific applications like medical or industry, there may be some fixed parameters for an interface and the fixed parameters can be added into a new separate interface table.
In the text table of the OmniToken of physics pixel, we can add prefix and suffix to the text representation for a specific datafield (like for interface, object, etc) to label the start and the end of the text representation to differentiate from other text tables for text in other tokens.
In labeled training, we can add text representation of the content of an interface, the interface ID and object ID to the text tables of tokens, to let the model learn the correlation between the features of or content on the interface and its corresponding text representation. You can train model by purely 2D images or video which include text or graph with labeled text representation in the text table of tokens which include the visual data in its physics pixel table at the same time.
For example: in labeled training, the OmniTokens can be from a video of driving car with a roadsign in the video, and training data labels the surface of the roadsign as an interface and labels the text representation of text or graph of the roadsign, the interface ID and object ID in the text tables of the tokens for the video to train the model learn the correlation.
For another example: in labeled training, the OmniTokens are all images from flat media like books or papers, and label a page of a book as an interface and label the text representation, the interface ID and object ID in the tokens from the image to train the model learn the correlation.
By labeled training above (the training data for which is most easy to get, right?), the model can learn to OCR 2D content from any visuals or visual flows, haha.
So below I revised the token example in last post to OmniToke including interface ID for each physics pixel, but you can label unnecessary interface as void like interface ID = “0” in training to let model learn to generate valid interface ID only for those interfaces that matters. The below OmniToken also change the length unit of Z in XYZ of perspective coordinate system and the length unit of X’Y’Z’ in rectangular coordinate system from 32 bits of millimeter to 48 bits of micrometer, which makes it enough accurate for most applications, and keep the pixel ordinal number X and Y as 16 bits unchanged for 65000 is enough for any X or Y pixel ordinal number. By the way, the asymmetry in last post shall mean different formats not only between XY and Z in XYZ but also between XYZ, X’Y’Z’, direction in spherical coordinates, rotation vector, etc..
The OmniToken below is intended to be an example for Terminal AI like vehicle, robot or drone, and for Server AI the token may have much larger fields like 10x1028D for text table.
The OmniToken example below includes following data tables which all are orthogonal in different dimensions (here 1D=32bits).
——/global parameters for OmniToken/——
1D: token ID,
1D: timestamp,
1D: data source, (input/external raw or output/internal generated, synthetic or not, physics complied or not, etc)
2D: mask tensor, (for text, RGBA, RGB, etc)
——/1536D for text/——
TABLE HEADER {
8 bits: table ID, (“01”)
8bits: number of rows, (“1”)
16 bits: number of columns, (“1536”D)
}
1536D: text.
32D: vacant space. (to separate text field from other fields)
——/list for sensors in this token/——
TABLE HEADER {
8bits: number of rows, (“3”, 3 types include camera, mm wave radar and microphone, each token includes one sensor of each type)
16bits: number of columns, (“6”D)
}
for (each of “3” sensors in this token) {
8bits: T=sensor type, (“1” means camera)
8bits: N=sensor ID, (“1” for first camera)
8bits: C=coordinate system type, (“C0” the unified rectangular coordinate system)
3D: position of sensor, (position in above “C0” coordinate system)
2D: direction in spherical coordinates, (horizontal+vertical angles in above “C0” coordinate system)
16bits+16bits: resolution of seneor, (for camera, it’s like 16bits=1920, 16bits=1080 which means 1920×1080 pixel; for mmwave radar, it’s like 16bits=0, 16bits=300, which means 300 3D points generated)
}
8D: Vacant space.
——/table for physics pixel/——
TABLE HEADER {
8bits: table ID, (“03”)
8bits: rows, (64)
16bits: columns, (?)
}
for (each of 64 physics pixels) {
??D: TNC+XYZ, (C= “01” means sensor’s own perspective coordinate system, XYZ is the position of physics pixel in perspective coordinate system of the camera, X=16bits is physics pixel‘s horizontal ordinal pixel number (from 1) in the 2D frame of the camera, Y=16bits is physics pixel‘s vertical ordinal pixel number (from 1) in the 2D frame of the camera, Z=48bits is distance from the physics pixel to the camera lens in micrometer, and “Z=0” means raw pixel received by camera and “Z=1” means points on camera lens)
??D: RGBA, (RGBA at “X,Y,0” is for raw RGB, RGBA at camera lens interface is on “X, Y, 1” to be distinct from raw RGB, and for any invisible physics pixel RGBA=”-1,-1,-1,-1”)
??D: C+X’Y’Z’, (mapping 3D point of the physics pixel, C=”C0” unified rectangular coordinate system, X’, Y’ and Z’=48bits in unit micrometer)
??D: direction in spherical coordinates, (horizontal angle+vertical angle in “C0” system, which is the direction perpendicular to the interface which is between near object and far object and which the physics pixel is on)
??D: interface ID, (in labeled training this is labeled, in reference this is generated by the model)
??D: pressure,
??D: near object ID,
??D: selected parameters of the point of near object on the physics pixel, (this field is optional, which may include temperature, velocity, rotation vector, hardness, density, material or no parameter at all)
??D: Far object ID,
??D: selected parameters of the point of far object on the physics pixel.
}
8D: Vacant space.
/table for objects in this token/
TABLE HEADER{
8bits: table ID, (“04”)
8bits: rows, (4)
16bits: columns, (?)
}
for(i=1 to 4){
object_i ID,
object class,
parent object,
object mass,
object velocity,
object rotation vector,
object_A ID,
pressure with object_A,
object_B ID,
pressure with object_B,
object eatable or not,
}
8D: Vacant space.
/table for interfaces in this token – this table is Optional/
TABLE HEADER{
8bits: table ID, (“05”)
8bits: rows, (“4”)
16bits: columns, (?)
}
for(i=1 to 4){
interface ID,
object ID, (if features below are for boudary of both surfaces of the interface, object ID=”0″)
interface parameter 1,
……
}
8D: Vacant space.
/table for mm wave radar/
TABLE HEADER{
8bits: table ID, (“06”)
8bits: rows, (?)
16bits: columns, (?)
}
/each table includes 10 3D points of total points generated by this mm wave radar at one time/
For (i=1 to 10) {
TNC+x’y’z’, (T=”2” mm wave radar, N=”1” first radar of the type, C=”C0” unified rectangular coordinate system, x’y’z’ all are 48bits)
velocity vector.
}
8D: Vacant space.
/table for microphone/
TABLE HEADER{
8bits: table ID, (“07”)
8bits: rows, (?)
16bits: columns, (?)
}
{
TNC+Sound signal of 1/15 sec.
}
16D: Vacant space.
/table for control/
TABLE HEADER{
}
{
user set parameters,
internal sensors,
actuators.
}
16D: Vacant space.
/table for agent/
TABLE HEADER{
}
{}
1024D: vacant
Be First to Comment